Fix Azure-incompatible file paths in PyArrowFile #2683
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
hey @NikitaMatskevich maybe we should open an issue and move the discussion there :) Im not sure if i understand the underlying issue and what is not working. Could you provide some more details? |
3 tasks
|
Hi @kevinjqliu , thanks for looking into it! I copy-pasted the description to the issue: #2698 and added a concrete example of what happens and why it is surely a bug. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Rationale for this change
Starting from version 20, Pyarrow has support for Azure filesystems.
Azure table locations are typically of this format: "abfss://<bucket_name>@<account_name>.<dfs|blob>.core.windows.net//
/<file_path>". When creating a PyArrowFile, we simply retrieve table location and append table-relative path to it. This generates a path with "@<account_name>.<dfs|blob>.core.windows.net" part in it, which cannot be read/written by Pyarrow library. One has to truncate this part from Azure uris.The proposed fix is just to start a conversation around the issue. I am not 100% sure how and where this should be fixed.
We know this issue does not occur with Fsspec.
Are these changes tested?
Hard to test, because with Azurite it works fine (unlike "real" Azure, Azurite does not have this part in uris). Do you have any ideas of an integration test in mind?